Add new EF 11 features - #296
Open
ChrisJollyAU wants to merge 75 commits into
Open
Conversation
Probed ACE (OLE DB 16.0/12.0) at the counter boundary: there is no overflow error. The TDEF high-water (0x14) is a plain signed int32 and the next id is 0x14 + increment computed unchecked, so an ascending counter runs ... 2147483647, -2147483648, -2147483647 ... and a descending one mirrors it. ACE writes the wrapped id to 0x14 and carries on. A wrapped id that is already occupied is an ordinary duplicate-key rejection, and ACE burns the id anyway (0x14 advances despite the failed insert) so the next insert steps over it. LibRed generated the same wrapped id but then wedged: UpdateTdefCounters' monotone guard - the deliberate KB 884185 immunity - read the wrapped value as going backwards and left 0x14 pinned at int.MaxValue, so every later auto insert reissued int.MinValue. The damage was the on-disk 0x14, so ACE opening the file failed the same way. AssignAutoNumbers now returns a per-value flag array of the ids it generated, and UpdateTdefCounters takes a generated id as the new high-water unconditionally: it came from 0x14 + increment, so it is the next value in the sequence by construction, wrap included. The monotone guard now applies only to caller-supplied explicit ids, which is the only case KB 884185 was ever about. Confirmed that guard is still needed: making explicit ids leave 0x14 untouched instead fails 5 tests, including ACE reusing id 1 after LibRed bulk-writes rows 1-3 - explicit ids are how data gets into a counter column, and 0x14 is the only record of where the counter is. Also moves the AssignAutoNumbers doc comment onto the method (it had been orphaned above MaterializeLongValues). Spec: page-02a-tdef.md gains a verified wrap note under 3.1, and the 0x14/0x18 rows plus the appendix entry record that the value wraps. Tests: AceAutoNumberOverflowProbeTest covers both engines at both boundaries, the explicit-int.MaxValue route to the wrap, and the occupied-wrapped-id case. LibRed.Core 481/481, LibRed.Engine 898/898, LibRed.Ado 47/47. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Implemented JetParseTranslator to map .NET Parse methods (e.g., int.Parse, double.Parse) to Jet/Access SQL type conversion functions (CBYTE, CDBL, CINT, CLNG, CDEC). Registered the translator in JetMethodCallTranslatorProvider. Updated related tests to expect Jet/Access SQL syntax for Parse operations instead of SQL Server-style CAST/CONVERT.
Squashes a batch of engine fixes and the test work that found them. Engine and format fixes (LibRed.Core / LibRed.Engine / LibRed.Ado): - Empty Binary index key is the start flag alone (7F asc / 80 desc), not a zero-padded chunk. ACE oracle: ACE writes the index, LibRed reads the stored entry back and re-encodes it. A LibRed-written empty key previously could not compare equal to an ACE-written one in the same index. - Office-Standard EncryptionInfo is parsed from the declared frame (len@0x299, blob at 0x29B) instead of scanning page 0 for a signature. The length is authoritative for ACE itself - a file with key + descriptor but len = 0 is read by Access as plaintext - so the old scan opened databases ACE cannot. Verified by falsification: with the scan restored, LibRed opens such a file successfully. - UPDATE (Flag 4) and DELETE (Flag 5) stored action queries are recognised and reported as not-yet-executable, rather than falling into the generic unsupported bucket. Flags verified against ACE-authored procedures. - Row pointers are bounds-checked before decode, so a corrupt index entry raises InvalidDataException instead of decoding arbitrary bytes. - Commit validates every overlay page against the committed image it was derived from, under a per-file publish gate, so overlapping writers conflict deterministically instead of losing an update. A failed publication restores the already-published prefix and keeps the transaction rollbackable, reporting both the publish failure and any restore failure. - Schema-changing commits advance a per-file catalog generation; other open connections reload their parsed catalog on next access, while plain DML does not force a reload. - Function argument arity is validated against ACE's JES, including Jet quirks (two-argument IIf yields Null). Aggregates go through the same contract. - SQL COMMIT/ROLLBACK reconciles the ADO transaction handle. Page scope is now reader/writer rather than a mutex. A statement that cannot write (SELECT, set operation, system-variable select) takes it shared, so concurrent readers on one file still run together; everything else takes it exclusive for the whole statement, which is what makes a multi-page write atomic to readers. Anything not provably read-only takes the exclusive scope - the shared scope cannot be upgraded and says so rather than deadlocking. Parsing happens before the scope is taken. Test suite: - Wall-clock guards replaced with structural assertions (the planner is asked directly whether the rewrite engaged), and ThrowsAny<Exception> replaced with specific exception types plus message assertions. - Shared AceTestDatabase / TemporaryDatabase helpers, in test/LibRed.Shared so the EF functional projects (which glob test/Shared wholesale and do not all reference LibRed.Core) are unaffected - EFCore.Jet.FunctionalTests had stopped building. - Temp databases are released per test. A database opened and abandoned by a static Fresh() helper kept its file locked for the whole process, so the copy survived every cleanup path; ~22 GB had accumulated in %TEMP%. Handles are now owned by the helper, closed before deletion, and released when the test ends, with a process-exit backstop. Peak temp copies during an engine run: 609 -> 19. - The five ACE-driving classes share one xunit collection. Concurrent ACE use faults natively (SEHException, then 0xC0000005 kills the run): those classes alone crashed 3 of 3 back-to-back runs, the other ~950 tests were clean 3 of 3. Parallelism is not disabled - only those five are serialized, against each other. - New: SchemaVisibilityTests (cross-connection catalog freshness, previously untested), reader/writer scope tests, and a zero-length-descriptor guard. Each was verified to fail with its mechanism disabled. - The Access-output legacy password comparison is restored as a fixture-gated test that skips with a reason (LIBRED_ENCTEST_DIR) instead of silently passing. Docs updated: transactions.md (scope semantics), page-00-database.md (descriptor framing, password fixtures), page-03-04-index-btree.md (empty Binary key), system-catalog.md (action flags), functions.md (arity). LibRed.Core 599, LibRed.Engine 962, LibRed.Ado 47 - all passing, no crashes, zero temp files leaked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…m tables Access 2010 added a second "General" sort order, and the collation version byte selects a genuinely different weight table rather than being metadata: the same values indexed in a v0 and a v1 database differ in all 28 samples. v1's primaries are the Windows NLS (Script Member, Alphabetic Weight) pair verbatim - "apple" is 0E02 0E7E 0E7E 0E48 0E21 - where General Legacy compacts the same ordering into one byte per character. That is why v1 can be derived from a published table while v0 has to be measured. Also records what ACE needs MSysComplexColumns for, which is its own feature rather than part of the collation work. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…gaps are for Bytes 0x0B-0x0E (page-0 0x6E-0x71) are ONE Windows LCID with the sort-order version in its otherwise-unused top byte: LANGID at 0x0B, sort id at 0x0D, version at 0x0E. The sort id is what separates a Windows alternate sort order from its base locale - German Phone Book 0x00010407 against German 0x00000407 - and they differ in nothing else. The spec had warned for months that 0x0D was "0 in every file seen, keep an eye on it". The gaps in the v0 letter table are insertion slots for language letters. It steps by +2 everywhere except B-C, Q-R and X-Y, and Spanish lands on exactly the free value in each relevant gap. The second byte is a SUB-POSITION ordering letters that share a slot, proved by locales that put several in one. Version 1 is not General-only: Croatian and Romanian ship in both generations. Five of DAO's collating orders are dead metadata, byte-identical to General, so appearing in the UI does not imply an implementation. DAO can author a locale but not a sort-order version, which is what fixtures need Access itself for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
JetLocaleTailoring keys per-locale overrides by STRING rather than character, so a contraction works: a digraph weighing as one letter is the inverse of the existing expansions, and the one primitive neither encoder had. Matching is greedy longest-first with no backtracking, and looks up the ORIGINAL text before the uppercased text, which is what lets Turkish disagree with invariant casing. Only Hungarian doubles, and that test must run before the longest match. Tailoring is not only insertion. Six devices, all inside the existing framing: insertion, contraction, expansion, secondary retune, remapping the base table, and reordering. Empty tailorings mean "measured to need no change", which is different from having none. Also fills in the General diacritic table and fixes the long s, which is a letter of its own rather than a fold onto s - found by testing each locale against a set far wider than its own tailoring. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…venance v0 IS the NT4-era NLS order, renumbered into one byte, and the compaction is ORDER-PRESERVING. Sorting every character by the Windows NT 4.0 - Server 2003 table's (SM, AW) primary and checking v0's bytes come out non-decreasing keeps 507 of 510 strictly-ordered pairs and 947 of 955 ties, with 12 of 14 blocks perfect. So the +2 stride, the language-letter insertion gaps and the 0x79 page are one decision rather than three observations. Jet also NARROWED it: 88 of the 552 v0 ignorables are weighted by NLS and dropped anyway, an editorial call no published table would reveal. Locales SHARE the block tables, with per-locale deltas in their own tailoring, which is what makes 21 orders cost 27 entries between them. A ligature weighs as its DECOMPOSITION - there is no ligature mechanism in the format. Components are weighed individually and never re-enter the contraction matcher, and decomposition sits below the tailoring because some locales do not decompose at all. The last gap was the word-sort ignorables, 20 of them rather than 3: every dash, the Arabic harakat, and fullwidth apostrophe and hyphen, which share their ASCII counterparts codes exactly - the one place fullwidth really does collapse onto ASCII, unlike the letters. Coverage is 2147/2147 for all 23 orders. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Prefix compression covers the WHOLE entry including its trailer, not just the key bytes. Reading it as key-only worked until a page held many equal keys, which is exactly what a full-BMP sweep produces - thousands of ignorable characters all encoding to the same empty key - and then the reader rejected the page outright. Found by the probes added here rather than by a test written for it, which is the argument for sweeping a whole range instead of sampling it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every character ACE stores a key for, LibRed now encodes identically - 63,422 of them. No published table describes v0, since its primaries are a Jet compaction rather than the NLS weights, so ACE ITSELF is the source: SortKeyTableV0GeneratorTest inserts every code point into an indexed text column, reads the stored keys back and writes the embedded resource (74 KB; 63,105 weights, 40 word-sort ignorables, 276 kana). Far past anything hand-maintainable, and hand-transcribing hex is exactly the work that introduces a wrong byte nobody notices. Two things only a full sweep shows: ACE weighs every CJK ideograph and the entire private-use area, and across all 65,536 code points it refused exactly one. KANA take a two-byte primary 7F <sound>, with voicing as an ordinary secondary and the small/normal distinction bit-packed into a section of its own - three per byte, two bits each, most significant first, under a 10 marker. Two rules only multi-character strings reveal: the halfwidth voicing marks are COMBINING (alone they look ignorable, which is what hid it), and the inline section's introducer becomes FF 01 when a kana section precedes it. The prolonged sound mark lengthens the preceding kana's VOWEL, which is what the character means - がー is "ga" lengthened by "a", not by "ga" - so the vowel is a property of each kana and has to be measured per character rather than derived. Also: inline positions count primary WEIGHTS, not bytes, and the hand-verified expansions stay ahead of the measured table because a key cannot show whether two bytes are one weight or two. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
v1 now encodes all 63,422 BMP characters exactly as ACE stores them, matching v0. It was 29,776 differences when the sweep started. Most of it was four structural rules rather than missing weights, which is why the sweep was worth running before generating anything: a Han character takes a FOUR-byte primary (the marker FD FF then its own weights) and no secondary, which alone was 28,200 wrong keys; Hangul jamo take (AW, DW) as the primary with no marker; a zero alphabetic weight means NO primary at all; and where something precedes it, such a weight FOLDS into the one before rather than taking a slot. The rest is that the published Server 2008 table is not quite what ACE carries. That identification came from 25 reconstructed keys, all Latin and symbols, and it holds for 57,793 characters and fails for 501 - Balinese and Canadian syllabics get Latin weights, and the Arabic harakat and several ligature blocks differ. Scripts added or reweighted since. Rather than hunt for the right NLS revision, the disagreements are measured and embedded (2.0 KB), along with 5,082 characters ACE treats as wholly ignorable that the published file has no entry for at all. An override stores raw primary and secondary bytes, not (SM, AW, DW) weights: that reading assumes a two-byte primary carrying one secondary, and ACE breaks it both ways - the harakat have a secondary and no primary, the Lao vowels take a one-byte primary. A primary byte can also BE 0x01, so the section delimiter is the last 0x01 in a key rather than the first. Splitting at the first made five characters look like an unknown mechanism; measuring them in combination showed ordinary two-weight expansions. Kana turn out to be shared: same sound weights, same section, byte for byte under both orders, so JetKanaSection is extracted rather than duplicated. Two narrow differences remain - v1 weighs a compatibility form by its base kana's sound where v0 gives it its own, and five kana absent from the v0 table come from v1's own script member 3, whose smallness cannot be inferred from reaching that path. A generator must run with its own resource suppressed. It records where the encoder DISAGREES with ACE, so measuring an encoder that already consults it would find no disagreements and write an empty file. Behaviour change: an unassigned character such as U+0378 no longer throws. ACE stores an empty key for it, so refusing would reject a value the engine accepts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ACE stores an index entry of at most 510 bytes as built. At exactly 510 it comes back byte-for-byte; a value needing 511 comes back as 510 with the weights cut short and the last two bytes replaced by a value that varies with the string - a truncated key plus a checksum, which is why two long values never collide. The checksum function is not known, so LibRed cannot reproduce a truncated key and refuses the value instead. It was writing the full-length key, which put bytes in the index ACE would never write, and a wrong index key is silent: ACE writes its own into the same index and a seek misses rows. The cap is on the whole ENTRY, not per column. Two 200-character text columns weigh about 404 bytes of key each - comfortably under the cap individually - and ACE stores their combined entry hashed at 510. A per-column check would have let that through. Because it limits weights and not characters, what it buys varies with collation and script, which is the practical cost of General over General Legacy and is invisible in the schema: 255 characters for v0 Latin (the column limit is reached first), 253 for v1 Latin, 169 for v1 accented, 127 for v1 Han. Above the BMP the two orders disagree completely, measured over all of planes 1 and 2 and sampled across all sixteen. v0 ignores astral characters entirely - every one gets the empty key - so under General Legacy an astral character is invisible to the index. v1 weighs BOTH surrogate halves, each looked up like any other character: U+10000 is 7F B002 B4F8 01 3F 3F 00. Only the high surrogates to U+D87F carry weights, so from plane 3 upward the high half is ignorable and the low one stands alone, and those planes collapse onto 1,024 keys. The fix v1 needed was therefore narrow: an unweighted surrogate is ignorable rather than an error. The tempting reading of the plane-3 samples - "the high surrogate contributes nothing" - is wrong, and skipping every high surrogate breaks all 131,068 characters of planes 1 and 2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… plus a byte An apostrophe or hyphen carries no primary weight and instead appends a record holding the position it sat at. That position is SIXTEEN bits, big-endian, with bit 15 set - [MS-UCODEREF] gives SpecialWeightType as (Position: 16 bit integer, ScriptMember, PrimaryWeight), emitted as "Byte1 = Position >> 8, Byte2 = Position & 0xff". The 0x80 is not a marker byte at all. LibRed read it as a marker followed by one position byte and truncated the rest. The two readings agree below 0x100 and diverge above it, and the offset 0x07 + 4 x position passes 0xFF at position 62 - so a hyphen at character 63 is 81 03 where LibRed wrote 80 03, and at 250 it is 83 EF where LibRed wrote 80 EF. Every indexed value with an apostrophe or hyphen past character 62 therefore got a wrong key. A hyphenated name in a 255-character column is enough, and nothing caught it: single characters encode correctly, short strings encode correctly, and the field only overflows when a value is long enough. It surfaced while reverse-engineering something unrelated. The lesson is in the spec beside it - measure combinations, not only characters, because a per-character sweep can be exhaustive over all 63,422 and still miss a whole class of bug. Measured against ACE across positions 10 to 250 under both sort orders. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…values Past 510 bytes ACE keeps the first 508 and replaces the rest with two bytes derived from what it dropped, which is why two long values sharing a prefix still sort apart. LibRed refused those values because the function was unknown. It is known now, and they are truncated exactly as ACE truncates them. Recovered by measurement rather than found documented. Three tails differing in one byte show the function is affine over GF(2) - L(0xA3) ^ L(0x13) = L(0xB0) exactly - and 173 observations show it is shift-invariant, so a byte at distance d from the end contributes S^(d-1) of itself whatever the length. Sweeping all 65,536 polynomials in five framings found NOTHING, and that negative was the clue. The standard reflected update is crc = (crc >> 8) ^ T[(crc ^ b) & 0xFF], passing the byte through the table; ACE computes crc = (crc >> 8) ^ T[crc & 0xFF] ^ b and injects it raw. Wrong injection point, so no polynomial could ever have matched. The step operator then came out of Gaussian elimination over the measured contributions and predicts all 657 of them - no name for the algorithm required. There is no initial value and no final XOR. The limit is on the whole ENTRY, not per column, which the measurement also settled: two 200-character text columns weigh about 404 bytes of key each, comfortably under the cap individually, and ACE stores their combined entry truncated. A per-column check would have let that through. Still refused where the dropped bytes hold an inline word-sort record. That cannot be verified even in principle - the record sits in the part ACE discarded, so what it contained is unobservable, and if ACE recomputes its position when truncating then the checksum's input is not what is reconstructed here. Guessing would write a silently wrong key. Text columns now index to the full 255 characters again rather than being refused past 127 to 253, depending on collation and script. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
French tailors no letter at all. It is General with the diacritic section written BACKWARDS, so accents are weighed from the end of the word and cote < cote-with-acute-on-e ... in short, coté sorts before côte where General has it the other way round. [MS-UCODEREF] names the flag IsReverseDW and gives both halves of the rule: the run of default diacritics comes off the LEFT rather than the right, and what remains is written right to left. Verified against ACE byte for byte. côté is [02 12 02 0E], trimmed to [12 02 0E] and stored as 0E 02 12. Across all of Latin-1 and Latin Extended-A with accents doubled and tripled per string, 1,289 values, zero differences. It had been recorded as "unclassified, secondary-section tailoring", which described the symptom rather than the rule, and the reason is worth keeping: a word with ONE accent encodes identically under both orders, and the sample set that measured every locale against General contained no two-accent word. The rule was invisible to the measurement rather than absent from it - the same blind spot as the inline position field. LibRed can now also CREATE a French database, which follows for free: creating one requires encoding the order, because the system-table indexes are built on the way. That circularity is why measuring French needed DAO to author the file first. ACE indexes into a LibRed-created French database with every key identical to LibRed's own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The script members were derived by measuring ACE, one class at a time, and the GetWindowsSortKey pseudocode names every one of them: PUNCTUATION 6 is the word-sort class, JAMO_SPECIAL 4 the Hangul jamo, EXTENSION_A 5 the Han shape, NONSPACE_MARK 1 the "no primary, only a secondary" rule. Everything at or below MAX_SPECIAL_CASE goes to its SpecialCaseHandler - which is exactly the set of classes that needed bespoke handling here. Script member 3 is EASTASIA_SPECIAL, not "kana", so the constant is renamed. That also turns an unexplained list into one rule: the class reserves PW_REPEAT 0 and PW_CHO_ON 1, and the seven characters ACE gives the unweighted FF FF primary are exactly those - the iteration marks and the lone prolonged sound mark. The 01 01 01 before a word-sort record is not an introducer but three SECTION SEPARATORS. The frame is primaries 01 diacritics 01 case 01 extra 01 specials 00, and Access emits it with the case-weight section EMPTY - which is the mechanism behind case and width folding, since width is bit 0 of the Case Weight. MIN_DW = 2 is the default secondary whose trailing run gets trimmed. Three further notes recorded in the spec. The contraction limit corroborates v0's provenance independently - 2 and 3 characters on NT4 through Server 2003, 4 to 8 from Vista, and every v0 tailoring here tops out at three, which is the same generation the weight-table comparison identified by a different route. The FD FF Han primary is NOT the Windows 7 three-byte weight, which is three bytes and postdates the table Access froze. And Access PACKS the East Asia extra weights three flags to a byte where Windows uses one byte per character. Nothing in that source covers the 510-byte cap, truncation or the checksum: a useful negative, since it means those are Jet inventions that had to be measured rather than looked up. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
These three were refused on the grounds that "the v1 encoder has no tailoring
hook: its primaries are 2-byte NLS values, a different shape". That was a fact
about the encoder, not about the orders. Measured, each is General v1 plus a
small override set using the same six devices every version-0 locale uses, and
all three are byte-identical to each other - the same order under three LCIDs,
so one table serves them.
The letters land where the Croatian alphabet puts them: L 0E48, LJ 0E4A, M 0E51;
D 0E1A, DZ-digraph 0E1D, D-with-stroke 0E1E. So the three digraphs are
contractions, which is why the loop had to become indexed. TailoredWeight
already carried byte[] primaries, so a two-byte primary needed no change at all.
Two things the conformance range had to be WIDENED to find, and both would have
written silently wrong keys for ordinary Croatian text:
- The caron retune reaches further than a hand-picked list of letters showed.
Eight more, and one of them moves its PRIMARY rather than only its accent.
- Expansion components were going straight to the base table, so the
precomposed digraph U+01C4 encoded as D + Z-with-caron instead of D + the
tailored Z-with-caron. Components take the LOCALE's letters - the rule
version 0 already followed - but must not re-enter the contraction matcher,
or expanding a ligature could trip a digraph the original text never had.
Version-1 fixtures were asserted over 447 values where version-0 ones got 2,444,
because the extended blocks were once measured for v0 only. That is no longer
true, and the narrowing only hid ground: it is removed, and all 27 fixtures now
run the same 2,444 values with zero mismatches.
One genuine ACE asymmetry recorded: U+016C takes the retuned secondary while
lowercase U+016D keeps General's, identically in all three locales. Every other
letter is case-symmetric, all three digraphs included.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…n 0) General v0 gave a secondary-only character a slot of its own in the secondary section. ACE folds it into the preceding weight instead, adding its value: Thai ไก่ is two weights with secondaries 03 06 - the tone mark's 03 added to the consonant's 03 - not three weights. Emitting a slot desynchronises the whole section from the primaries, so everything after the mark is wrong too. This affects ALL 28 orders, not one. The class is "secondary-only" characters, and it holds Thai tone marks, every Hebrew niqqud, the Cyrillic combining marks and three Greek ones - any indexed text with a combining mark following a base character. The version-1 encoder has had this rule since the full-BMP work; version 0 never got it. It surfaced only when the Thai block entered the conformance range, because the rule needs a mark AFTER a base character and a per-character sweep cannot place one there. Nor could comparing one locale against another: both were wrong identically, so only comparing against ACE shows it. The conformance range gains the Thai block and words, which is what caught it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Thai writes the five vowels e/ae/o/ai/ai-maimuan BEFORE the consonant they are pronounced after, and collation follows speech. That was recorded as needing REORDERING, a device nothing else here uses, and it is why Thai stayed unimplemented as the last non-CJK order. Measurement says it is an ordinary contraction - the same device Croatian's lj uses. ACE gives the pair a SINGLE weight at the consonant's own primary plus a vowel offset: the pair with ko-kai is 7C99 where the consonant alone is 7C98, the next vowel gives 7C9A, and so on to +5. Every consonant sits on a six-wide block, itself plus a slot per leading vowel. And it is a contraction rather than a swap, because the reverse order does NOT collide: consonant-then-vowel stays two weights, 7C98 7C93. Built as the rule rather than 220 transcribed entries, with each consonant's primary read from the measured v0 table, so there is no hand-copied hex. Also re-verifies the five DAO-only orders that are recorded as inert. They were established over 31 samples of single characters, and French proved that shape of evidence can hide an entire rule - it tailors no letter at all, so a word with ONE accent looks identical to General and only two reveal it. Now 82 samples including words carrying two marks per script, and the Greek triple that is the direct analogue of the French one. All five are still inert, and the probe's positive controls still show their departures, so the null result means the orders are inert rather than the harness being dead. Every non-CJK sort order is now implemented: 28 of them, each verified against ACE over 2,559 values. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI runs LibRed.Engine.Tests on five platforms - Linux, Windows, macOS and both ARM legs - because LibRed is fully managed, and that run is what actually proves the cross-platform claim. Twenty-four tests in there open an ACE OLE DB connection, so they fail on four of the five, and for a reason that says nothing about the code: the driver is simply absent. They move to a new LibRed.Engine.AccessTests, which runs under the LibRedAccess job beside LibRed.Core.Tests, where the other comparisons against the real engine already live. They genuinely need the engine - QueryEngine, SQL, the lot - so LibRed.Core.Tests could not host them without inverting the layering, and LibRed.Ado.Tests would have meant filing engine tests under the ADO layer and introducing an ACE dependency to a project that has none. The five classes were already marked [Collection(AceCollection.Name)], so they identified themselves and the split needed no judgement about which were which. LibRed.Engine.Tests now has no System.Data.OleDb reference and no AceTestDatabase, so this cannot drift back: an ACE test added there does not compile rather than failing in CI on four platforms. 938 + 24 = 962, the count before the split. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
LibRed synthesises a new .accdb page by page rather than copying a packaged empty file, so the collating order is a parameter of creation rather than a property of a template. That had been demonstrated for the two General orders and French; the other 27 configurations were an inference from sharing a code path. Now measured: 30 configurations - both General orders and every locale in JetLocaleTailoring, including the two orders that exist at a second sort id (German Phone Book, Hungarian Technical). For each, LibRed creates the file, ACE creates a table and index INSIDE it, and every key ACE writes matches LibRed's own. Nothing disagreed. The bar is deliberately that rather than "the file opens". Two engines agreeing on a shared index is the only check that catches a wrong key, because a disagreement does not error - it makes seeks miss rows. The list comes from asking IsIndexKeyEncodable rather than from a hardcoded set, so it cannot drift out of step with JetLocaleTailoring and a new order is covered the moment it lands. It also guards the entanglement between the two: the system-table indexes are built during creation, in the database's own order, so creating a database REQUIRES encoding its collation. A locale with wrong weights would not merely sort wrongly - it would make creation itself produce a file ACE disagrees with. That circularity is why measuring a new locale for the first time needs DAO to author the file. Also corrects the CreateEmpty doc, which still said only the two General orders could be encoded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Access has two append forms. The single-record one takes VALUES and was already
supported; the multiple-record one takes a query and was not, which is the only
way to append more than one row in a statement - Jet has no multi-row VALUES
syntax at all, so "many rows" and "from a query" are the same feature there.
INSERT INTO target [(field, ...)] SELECT [source.]field, ... FROM tableexpression
The IN externaldatabase clause both forms allow is deliberately left out:
appending into another file belongs to the linked-database subsystem LibRed does
not have, and a half-implementation would be worse than none.
Two behaviours were measured against ACE rather than reasoned, and one of them
caught a bug that all nine of my own tests had agreed with:
WITHOUT a column list, ACE resolves the source's output NAMES against the target
- not positionally, which is what this first implemented from the plausible
premise that only the count matters. The case that separates them is reversed
aliases: SELECT B AS Name, A AS Id emits ('seven', 7) in that order, and ACE
stores Id=7. Positionally it would have stored 'seven' in Id. ACE also rejects a
name the target lacks, SELECT * included, and LibRed now gives the same error.
WITH a column list the other rule applies: the list names the targets and values
map positionally onto it, whatever the source calls them.
Appending a table to ITSELF terminates. The source is materialised before a row
is written, or the scan consumes its own output forever; ACE doubles the table
and stops, and so does this.
The failure mode of getting the first one wrong is the silent kind - into
type-compatible columns it succeeds and puts the values in the wrong fields -
which is the argument for cross-checking rather than trusting green tests that
encode the same assumption as the code.
No cross-check exists for column DEFAULTs on the ACE side: its DDL rejects
DEFAULT in CREATE TABLE ("Syntax error in field definition"), a column property
Access sets through DAO/ADOX instead. That is a limit on what can be compared,
not a place the engines differ, and it is recorded in the test file.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
SELECT field1[, field2[, ...]] INTO newtable [IN externaldatabase] FROM source
A make-table looks like a SELECT and is an action query: it writes a table and
returns no rows. It runs with the writing statements, inside the implicit
transaction, so a failure part-way cannot leave a half-populated table.
Measured from ACE before implementing, because most of it could reasonably have
gone the other way:
- A make-table copies DATA AND COLUMN DEFINITIONS ONLY. The source's PRIMARY
KEY and its indexes are NOT copied, so archiving a keyed table gives an
unkeyed copy.
- A result column that IS a source column keeps that column's definition,
width included: a source Text(30) arrives as Text(30). A COMPUTED column has
no declared width to copy and gets Text at the 255 maximum.
- An empty result still creates the table.
- An existing target is an error - the docs call it "a trappable error" and
ACE says "Table 'X' already exists".
The width rule cost a round trip worth recording. The probe output showed both
cases - a named column at Text(60) and a concatenation at Text(510), two lines
apart - and this generalised from the second while looking at the first. So the
cross-check against ACE caught a misreading of a measurement, not just an
unmeasured guess.
Two routing points the page cache and the tests found rather than review:
ExecuteQuery reaches ExecuteQueryCore without passing Route, so a make-table
invoked that way returned the source's rows; and Scoped classified any
SelectStatement as read-only, so the table write was refused by the guard that
rejects a write in a shared scope rather than silently upgrading it. Both now
account for INTO.
IN externaldatabase is not implemented, as on INSERT: creating a table in
another file belongs to the linked-database subsystem LibRed does not have.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MigrationsInfrastructureLibRedTest never completed: the class hung, and a killed test host kept a handle on the .accdb so the next run could not delete the file either. Diagnosed by sampling the stuck process — every thread was parked in Thread.Sleep inside AcquireDatabaseLock — and then by instrumenting the loop, which showed it spinning on a returned 0 and never on an exception. It was not a deadlock. The retry delay started at 1s and doubled while under a minute, so after seven misses every contender retried once per 64 seconds; the delay never reset, and there was no jitter, so the threads woke in lockstep and exactly one won per round. Fifteen threads therefore took ~16 minutes, which is indistinguishable from a hang. Having no timeout, the loop could not fail, only wait, so nothing was ever logged. - Retry policy: 50ms start, 1s cap, +/-25% jitter, and a one minute deadline that throws a TimeoutException naming the lock table and how to clear it. Build the lock object on success rather than allocating and discarding one on every attempt. - AcquireDatabaseLockAsync was missing both guards its synchronous twin has: no catch around the racy lock-table CREATE, none around the insert's duplicate-key race. Mirrored. Fixing the wait exposed a second defect it had been hiding. Concurrent migrators all pass the non-atomic exists-then-create check and all issue CREATE TABLE; EF catches the losers as DbException, but LibRed threw InvalidOperationException, which escaped that guard and failed the migration outright. ACE raises OleDbException there, so translating is what makes LibRed behave like the engine it stands in for. - New SchemaObjectExistsException, deriving from InvalidOperationException as ConstraintViolationException does, thrown from the four DDL name collisions: CREATE TABLE, CREATE VIEW/PROCEDURE, SELECT INTO and ALTER TABLE RENAME. LibRedCommand translates it into LibRedException with ObjectAlreadyExists (2714). - Assertions on those paths now name the exact type, since Assert.Throws does not accept a derived one. - New tests pin the lock contract that had none: the statement itself through the engine, the acquire/release cycle through ADO, and N connections contending, all bounded so a regression fails instead of hanging. MigrationsInfrastructureLibRedTest goes from hanging to 34/36 in 84s. The two remaining failures assert SQL Server baselines (sp_getapplock, CREATE DATABASE, brackets) and were never ported. Engine 961/961, Ado 55/55, Core 789/789, Engine-ACE 32/32. JetHistoryRepository is shared, so the ACE path gets the same retry policy and async guards; that has not been exercised against a real driver. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
migrationBuilder.Sql("--Before") sends a command whose entire text is a
comment. The lexer skips WS/LINE_COMMENT/BLOCK_COMMENT outright, so that
text produces no tokens, and the statement rule has no empty production —
so it failed with "mismatched input '<EOF>' expecting {SELECT, IF, ...}".
Any user migration carrying a comment-only command hit this.
A comment is not a token, so this is not a missing statement kind and gets
no AST node: text with no tokens simply has no statement to run. The check
is answered by the lexer rather than by scanning for '--', because in
SELECT '--' the dashes belong to a string literal and a textual strip
would reduce a real statement to nothing.
- ISqlParser.IsStatementless, implemented by pulling one token and asking
whether it is already EOF.
- QueryEngine.ExecuteQuery/Execute short-circuit to an empty result. They
take no page scope: there is no work to isolate. Execute reports zero
rows affected rather than a query's -1, a comment being an action that
did nothing rather than a result set.
- LibRedCommand.ExecuteBatch skips such a fragment rather than running it.
ExecuteBatch returns the LAST statement's result, so without this
"INSERT ...; -- done" would report the comment's zero rows in place of
the insert's, and take @@rowcount with it.
Found in MigrationsInfrastructureLibRedTest, which died on the comment
before reaching anything it was testing. That test still fails, now on the
next command: its migration body is unported T-SQL (IF OBJECT_ID, THROW
65536, brackets), as is its baseline, and it fails on ACE too — it is
absent from the Jet green list.
Engine 970/970, Ado 59/59, Core 789/789, Engine-ACE 32/32.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reading MSysAccessStorage from the Northwind ACCDB failed outright: InvalidDataException: A relocation source must contain exactly one 4-byte pointer; found 55 bytes RowRelocationReader required a relocation slot to be exactly 4 bytes wide. That table has live overflow slots of 45-63 bytes, and the pointer is sitting in the leading 4 bytes of each one. The remainder is the row as it was BEFORE it moved, with only those 4 bytes overwritten. Discount them and every field lands where the row format puts it (the pointer covers the 2-byte column count plus the first 2 bytes of the first column, leaving that column's remaining 6 bytes at offset 4); the remnant's Id/ParentId/Type/Name equal those of the row it forwards to; the remnant's null bitmap differs from its target's in exactly one bit, the OLE column Lv, whose arrival grew the row and forced the move; and every remnant is shorter than its target. The slot kept the old row's width instead of being trimmed. So the width was never the contract - the leading pointer is. Require at least 4 bytes and read sourceBytes[..4]. The checks that actually validate a relocation are untouched and unchanged: the target must be in the file, owned by the same TDEF, and a nonempty hidden inline row. What writes the wide form is NOT known, and is deliberately not claimed in either the comment or the spec. Every writer reachable from code trims to exactly 4 - measured across 317 relocations with no exception, covering ACE on x64, the ACE 2010 runtime on x86, and LibRed's own writer, under growing and shrinking text, repeated re-relocation, page fragmentation by interleaved deletes, and an OLE column going from NULL to a value. Access's own maintenance of its system tables is not reachable through SQL DML and remains unexercised; the spec records that as the open avenue rather than guessing. Because no write path produces the shape, the wide case is covered by handing the resolver a wider source span directly rather than manufacturing it on disk. A companion test pins our own writer still trimming, so if that ever changes the spec's claim fails loudly instead of going stale quietly. page-01-data-and-rows.md said "contains exactly one 4-byte pointer" and "validates the exact source width". Both corrected, with the evidence. Core 793/793, Engine 970/970, Ado 59/59, Engine-ACE 32/32. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
JetByteArrayTypeMapping built its literal by appending "0x" and then each
byte as hex, so an empty array produced the bare string "0x". That is SQL
Server's empty-varbinary literal and it is not valid Access SQL: ACE answers
"Syntax error in query expression '0x'". The SQL was therefore wrong for both
providers - it failed on ACE exactly as it failed on LibRed, whose parser was
right to reject it (HEX_LITERAL already requires at least one digit, matching
ACE).
Verified against ACE, an empty STRING is the literal that works, in a value
position and in a DEFAULT: '' stores and reads back as a zero-length byte[],
and stays distinct from NULL (IS NULL does not match it). 0x00 is NOT the
same thing - that is a one-byte zero.
The parity half is the larger part. LibRed's codec cast straight to byte[],
so any string reaching a binary column threw InvalidCastException - not only
the empty one. Fixing just the empty case would have left 'A' broken.
ACE's rule, measured for VARBINARY and LONGBINARY alike, is that a string in
a binary column stores its UTF-16LE bytes:
'' -> byte[0]
'A' -> 4100
'AB' -> 41004200
'e' -> E900 (U+00E9)
'41' -> 34003100 the digits '4','1', NOT the byte 0x41
The empty case falls out of that rather than being special. JetTypeCodec now
routes Binary and Ole through AsBinary, which encodes a string that way and
passes a byte[] through untouched - the same UTF-16 treatment Memo text
already had.
Tests cover the rule, not just the fix: empty stores zero-length, empty is
not NULL, 0x00 is a one-byte zero, the four UTF-16 cases, hex still round
trips, and a digitless 0x is still rejected as ACE rejects it.
Spec check (LibRed.Core type-codec change): no docs/format update. The stored
representation is unchanged - raw bytes either way; what changed is which
input types the codec accepts. ACE treating binary as a Unicode string
throughout its SQL surface is worth documenting on its own terms once
explored (Len/LenB, comparison, concatenation), not as a footnote here.
Engine 979/979, Core 793/793, Ado 59/59, Engine-ACE 32/32.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A binary column has two faces in Access SQL, and LibRed implemented only one.
Measured against ACE:
=, <, >, ORDER BY BYTES - case-sensitive, byte order
LIKE, Len, &, TypeName a UTF-16 STRING, and LIKE carries the text
collation with it
so `B = 0x4100` matches only 'A', while `B LIKE 'A%'` matches 'A' (0x4100)
AND 'a' (0x6100) - the same column, compared case-sensitively by '=' and
case-insensitively by LIKE.
LibRed had the byte half exactly right already. The text half called
ToString() on the byte[], so it operated on the literal string
"System.Byte[]":
Len(B) 13 for every value ("System.Byte[]".Length) -> now 1
B & 'x' "System.Byte[]x" -> now "Ax"
B LIKE 'A%' 0 rows -> now 3
TypeName(B) "Byte[]" -> now "String"
Silent nonsense rather than an error, which is worse than the
InvalidCastException fixed in d2264a0 on the write side.
The helper was already there: ToText() reinterprets a binary value as UTF-16
and was written for the byte functions (LenB, MidB, InStrB - which is why
LenB was correct all along). This just routes the text functions, both concat
operators, LIKE and TypeName through it as well.
TypeName now reports String for a binary column rather than the CLR type
LibRed actually holds. That is deliberate: ACE's expression service sees a
VT_BSTR there, and VarType already returned 8.
Verified end to end against ACE - Len/LenB 1/2 and 0/0 for empty, = 0x4100
matching one row, ORDER BY putting 0x4200 before 0x6100, LIKE 'A%'/'a%'/'A_'
matching 3/3/1, 0x0102 & 'x' giving "ȁx", TypeName String and VarType 8.
Every one of those is now a test.
Engine 989/989, Core 793/793, Ado 59/59, Engine-ACE 32/32.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ByteArrayTranslations.Any failed on both providers with "The LINQ expression ... .Any()' could not be translated", while its three siblings - Length, First and Index - fail with the provider's deliberate refusal: Returning the exact length of a byte array is not supported by Jet ... There is support for a 'EF.Functions.ByteArrayLength' method ... Those three are correct. LENB reports the UTF-16 byte count, so it rounds an odd length UP to even and an exact length is genuinely unobtainable; ByteArrayLength offers the LENB / ASCB(RIGHTB(x,1)) workaround and documents precisely when it is wrong (data ending in 0x00, where a real even-length value is indistinguishable from a zero-padded odd-length one). Refusing beats returning a wrong number. Any is different: it asks only whether there are any bytes at all, and that question the rounding cannot spoil. An empty array is 0 and every non-empty array is at least 2, so LENB(x) > 0 is exact - the trailing-0x00 ambiguity cannot arise for a > 0 test, and unlike ByteArrayLength this needs no caveat. Both baselines were still SQL Server's - DATALENGTH([b].[ByteArray]) > 0, square brackets and all - i.e. copied and never ported, which is why nobody noticed the translation was missing. Now: WHERE LENB(`b`.`ByteArray`) > 0 Verified on both providers, the Jet one against a real ACE driver: 9 tests, 6 passing where 5 passed before, and the same three by-design refusals. A full functional run was not done; the new case is guarded on a parameterless Any over byte[], so nothing else can reach it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The DATETIME2 and BIGINT tests added over the last two commits assume the installed ACE can create those columns. That holds on a developer machine running Microsoft 365, and does not hold on CI, which installs AccessDatabaseEngine_2016_x64 - Date/Time Extended needs ACE 17 (Access 2019+), Large Number ACE 16. So they failed there for a reason that says nothing about LibRed. My regression: written against the capability in front of me without guarding for an older engine. AceTestDatabase.SupportsColumnType asks the question once per type - CREATE TABLE with that column on a throwaway copy - and caches it; the affected tests Assert.SkipUnless on it with a reason naming the Access version required. No particular format version is needed for the probe, since ACE raises the file itself when a column demands one. It assumes an ACE that does not know a type name rejects it rather than silently coercing it to something else. That is reasonable - ACE is strict enough about these names to reject even DATETIME2(7) and DATETIMEEXTENDED as syntax errors - but it is an assumption, so it is written down at the probe: if a guarded test ever fails on an older engine instead of skipping, that is where to look. Unverifiable locally, since this machine has the newer engine and correctly does not skip (804 passed / 46 skipped, unchanged). The confirmation is CI turning 8 failures into 8 skips. Note this makes the DATETIME2 work unverified in CI rather than verified - green because it skipped. Installing an ACE 2019+/365 redistributable on the runner, or carrying both in the existing aceVersion matrix axis, would actually exercise it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…uest The LibRedAccess job in pull_request.yml runs Core, Engine.AccessTests, Ado and EFCore; push.yml was missing the Engine.AccessTests step, so those cross-checks against the real Access engine only ran on pull requests and a push could go green without them. Added verbatim from the pull_request version, comment included, so the two job definitions now match step for step. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Introduced settings.json to explicitly allow only certain Bash and PowerShell commands (mainly dotnet and git). Added pre-tool-use hooks to deny shell commands that edit source files or run Python, and to restrict file reading commands to scratchpad/log files. This enhances safety and control over shell usage in the project.
The test for sysname column mapping to varchar(255) and nullability was removed from JetDatabaseModelFactoryTest.cs and LibRedDatabaseModelFactoryTest.cs. Related references were also deleted from ace_2010_odbc_x86.txt and ace_2010_oledb_x86.txt.
Both had drifted. AGENTS.md was the worse of the two, still claiming 10.0.x / EF Core 10 / net10.0 and missing the OLE Automation heritage section entirely. Corrections: - LibRed is on master and builds as part of the solution; it was still described as living on a `libred` branch. - JetMigrationsSqlGenerator does emit ALTER TABLE ... ALTER COLUMN. The claim that it cannot was flatly wrong. - JetQueryTranslationPostprocessor now appends the identifier column to ORDER BY for deterministic tie-breaking, and reaches SelectExpression._identifier by reflection - worth knowing, since an EF bump breaks that at runtime. - The test tree listed 5 projects; there are 12. Each is now tagged [Windows + ACE] or [cross-platform], which is what decides whether it can run at all. EFCore.Jet.Tests is called out as empty - its files are <Compile Remove>d, so a green run of it is not coverage. - LibRed already does format version gating and auto-upgrade (JetVersion, AccessTypeMapper.RequiredVersion, JetDatabase.EnsureFormatAtLeast). BIGINT and DATETIME2 sit at different version bytes, 0x05 and 0x06, not the single ACE 16 threshold one might assume. The Jet provider has no equivalent and still maps long to decimal(20,0); that asymmetry is now stated so neither side gets "fixed" on an assumption about the other. New sections: the green-tests baseline (the gate is "did anything that passed stop passing", not the failure count), the CI job layout, the .docker images, tools/, and the shell-edit restriction that .claude/settings.json enforces. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The EnableMillisecondsSupport option and all associated logic have been removed from Jet and LibRed providers, including interface properties, extension methods, internal state, configuration, and tests. Code handling millisecond support for DateTime, DateTimeOffset, and TimeSpan in Jet databases has been deleted. Migration test data for millisecond precision is now commented out, and test assertions use interpolated strings for expected SQL.
…m EFCore.Jet
Add a cross-platform EntityFrameworkCore.Jet.Common assembly holding the
Jet-dialect services shared by both providers, and move the bulk of
EFCore.Jet into it: query translation and SQL generation, migrations SQL
generation, conventions, update, value generation, logging and JetStrings.
Storage and Scaffolding stay in EFCore.Jet, which remains Windows-only.
Common keeps RootNamespace EntityFrameworkCore.Jet so the moved types and
the JetStrings resource name are unchanged.
The DUAL table name moves off JetConfiguration (it had no readers inside
EFCore.Jet.Data) onto the new JetDualTable, so the shared generator and the
scaffolding factory that detects the table can both reach it without the EF
layer depending on an ADO.NET driver. It stays user-settable via CustomName.
LibRed.EFCore now references only LibRed.Ado and Jet.Common. It registers
its own service graph rather than calling AddEntityFrameworkJet and patching
it, and gets 1:1 copies of the types that must differ per provider: options
and options extension, convention set builder, history repository, code
generator and design-time services. DataAccessProviderFactory is gone from
the LibRed test surface.
Move the [SupportedOSPlatform("windows")] stamp out of src/Directory.Build.props
onto the projects that need it, which lets src/LibRed/Directory.Build.props fold
back in with no overrides.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ests LibRed's props set IsPackable=false and imported the repo-root props directly, so the NuGet job's pack loop already ran over the five LibRed projects and silently produced nothing. Import src/Directory.Build.props like every other src project instead: they now pack with SourceLink, symbol packages and the PackPdb switch, the same as the Jet packages. Only the documentation file is turned back off. EFCore.Jet.Common needed no change - it already packed. The NuGet job no longer depends on any other job. The code has been tested on the pull request and again by this workflow's own test jobs, so making the publish wait on a third pass only delays the package. src/EFCore.Jet.Common was in neither path filter, so a Common-only change ran no suite at all; it belongs to both, since both providers sit on it. The libred filter's EFCore.Jet and EFCore.Jet.Data entries go: nothing under src/LibRed, and no LibRed test project, references either one since the decoupling. push.yml was also missing LibRed.Engine.AccessTests, which pull_request.yml already had. Drop the EF1001/CA1416 suppression from LibRed.EFCore - it was there for the Jet internals the provider no longer extends. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…t services
Introduce LibRedSqlMode (Extended = 0, the default; Compatible) and store it on
the LibRed options as SqlMode, replacing the earlier boolean. Extended is what
you get when nothing configures it; Compatible is opted into, either through the
new UseLibRed overloads that take the mode or via UseSqlMode on the builder,
whose parameter defaults to Compatible so calling it turns compatibility on.
Every existing UseLibRed overload gains a mode-carrying twin, so the set stays
symmetric: the no-connection, connection-string, connection and
connection-plus-ownership forms, each in the non-generic and <TContext>
families. On the connection path the three-argument form delegates to the
four-argument one with contextOwnsConnection false, mirroring how the existing
two-argument form already delegates.
Anything whose behaviour differs per mode now gets a 1:1 copy in LibRed.EFCore
rather than the shared type branching:
LibRedQuerySqlGenerator copy of the Jet generator; its factory picks
between the two by SqlMode
LibRedUpdateSqlGenerator copy, still implementing Common's
IJetUpdateSqlGenerator so Common's batching
keeps working
LibRedMigrationsSqlGenerator copy
LibRedQueryTranslationPostprocessor copy that runs the ORDER BY lift only in
Compatible mode; its factory injects
ILibRedOptions to make that call
Each copy differs from its Jet original only in the namespace, the type name and
those gates - the bodies are otherwise byte-identical, so a diff against the
original stays readable.
With LibRed no longer using them, the Jet query translation postprocessor and
its factory move back out of Common into EFCore.Jet. Common keeps what both
providers still share on that path: JetLiftOrderByPostprocessor and
JetSkipTakePostprocessor.
The functional tests opt into Compatible explicitly, in ApplyConfiguration and
at the sites that build options directly, so they no longer depend on the
provider default. SqlMode is deliberately absent from the options LogFragment.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A copy of EFCore.LibRed.FunctionalTests that runs against LibRed's own SQL generator instead of the Jet one. Extended is the provider default, so the copy opts into nothing: every UseSqlMode call is removed, including the one in ApplyConfiguration, and LoggingLibRedTest goes back to passing relationalAction straight through. Assembly name and namespace are EntityFrameworkCore.LibRed.Extended.FunctionalTests so the two suites produce distinct fully-qualified test names - otherwise merged reporting and stack traces could not tell them apart. Test discovery finds the same number of tests as the compat suite. A full copy rather than shared source or a shared base with two concrete classes, for three reasons: the AssertSql baselines are inline in each test method and nearly all of them will differ once the Jet workarounds are stripped out; EF Core's own model is one project per target, each deriving from the shared specification-test bases with its own baselines; and EF Core's baseline rewriter replaces the whole AssertSql block, so anything cleverer than a plain single baseline gets flattened by it. The stores do not collide with the compat suite despite sharing names - each project writes them under its own bin/<Config>/<TFM>, which is what already keeps Jet and LibRed apart. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
LibRedQuerySqlGenerator no longer rewrites NULLIF(a, b) into IIF(a = b, NULL, a). That rewrite exists because ACE has no NULLIF - it answers "Undefined function 'NULLIF' in expression" - but LibRed's engine has implemented NULLIF since a5aed06, with its own tests in LibRed.Engine.Tests/NullIfTests.cs, so extended mode can emit the plain form. The extended suite's five Conditional_uncoalesce baselines already expect NULLIF, which is what they said before 0529e1c introduced the rewrite; they arrived that way with the suite in fa01fe3 and have been failing since, because the generator was still producing IIF. Removing the rewrite is what makes them pass. The compat suite keeps IIF, since it still runs the Jet generator. This is the first of the Jet workarounds to come out, and it was picked because the engine already supported it: the two suites now produce different SQL for the same tests off a single generator swap, which confirms SqlMode reaches LibRedQuerySqlGeneratorFactory and that each suite selects the generator it should. A failure from here on means the engine genuinely lacks something. Also removes GreenTests/ace_2010_{odbc,oledb}_x86.txt from the LibRed functional tests. The pass-lists belong to EFCore.Jet.FunctionalTests, whose matrix is ACE version x architecture x ODBC/OLE DB; LibRed uses no driver, so those legs do not apply to it and the files were strays. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
LibRedParameterBasedSqlProcessor is a 1:1 copy of the Jet one that runs two of
the passes in compatible mode only:
JetOuterJoinProjectionGuardExpressionVisitor wraps row-independent
projections inside LEFT JOIN subqueries in a CASE the Jet generator needs
JetCompatibilityExpressionVisitor throws "Unsupported Jet
expression" for ROW_NUMBER, CROSS/OUTER APPLY, EXCEPT, INTERSECT, JSON
paths and some join predicates
Both are Jet-dialect workarounds. The second is the more consequential one to
drop: it is a rejection pass, so skipping it does not change any SQL text, it
stops extended mode refusing the constructs LibRed exists to support. Expect
those tests to move from a clean translation failure to whatever the parser,
binder, planner or executor says about them - that list is the capability gap,
and it is meant to be worked through gradually rather than all at once.
The other passes are unchanged and still unconditional: JetZeroLimitConverter,
JetDateTimeRangeConverter and JetSqlNullabilityProcessor.
Its factory injects ILibRedOptions to make the decision, the same way the query
translation postprocessor's does, and takes over the
IRelationalParameterBasedSqlProcessorFactory registration.
With LibRed no longer using them, the Jet processor and its factory move back
out of Common into EFCore.Jet, joining the query translation postprocessor pair
that moved earlier. Two doc comments in Common referenced the moved type with
<see cref>, which Common can no longer resolve; since CS1574 is suppressed
repo-wide that would have rotted silently, so they are now plain <c> text, which
is what the surrounding prose already does for types it does not link.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
27 test files, from the first extended-mode run: 36234 passing / 1567 failing /
624 skipped / 19 not run of 38444, against the same-day compat baseline of
36119 / 235 / 2071 / 19. The arithmetic balances exactly - 1447 = 1332 + 115 -
so every delta is a previously-skipped test that now runs, and nothing that
passed in compat regressed.
These baselines only exist for tests that pass: AssertSql runs after the query
has executed and its results compared, so a test failing on either never
reaches it and the rewriter never touches it. Anything still failing keeps its
original placeholder.
What changed, by count across the diff:
412 [bracket] lines -> 0 upstream SQL Server placeholders in tests that had
never run, now real Jet SQL. The dominant category.
FROM ( 120 -> 17 nesting collapse from dropping the ORDER BY lift,
) AS 310 -> 136 which cost a derived-table layer per lifted
ordering. The largest structural change.
10 interpolated $""" -> 0 baselines that had been hand-written in plausible
backtick SQL for tests that could never execute;
3 of them said CHARINDEX where the provider
actually emits LIKE.
LEFT JOIN 77 -> 85 join predicates containing constants and
INNER JOIN 74 -> 67 parameters now translate instead of being refused
by JetCompatibilityExpressionVisitor, whose
ContainsUnsupportCol rejected any predicate
holding a SqlConstantExpression.
CROSS APPLY, OUTER APPLY and ROW_NUMBER appear nowhere in the diff, on either
side - none of those reached a passing baseline, so they remain the bulk of the
1567 and are the next real capability gap.
Worth watching in the speed phase: without the lift, the correlated subquery is
written twice, once in the projection and once in ORDER BY, where the lift
materialised it once as a column.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…alue constructor Four features, each one the plain SQL form that EF Core's base generator emits once a Jet workaround is stripped out of the extended-mode generator. They share the same files, so they land together. Engine suite 1104 passing, 0 failing. ANSI paging - OFFSET n ROWS / FETCH FIRST m ROWS ONLY / both. Access has no OFFSET at all, which is why the compat path emulates a page with nested TOP and a reversed ORDER BY. The three shapes come from GenerateLimitOffset: a bare Skip emits OFFSET, a bare Take emits FETCH FIRST, and together they emit FETCH NEXT. FIRST and NEXT are synonyms in the standard, as are ROW and ROWS, so both spellings parse either side; a grammar taking only NEXT would reject every plain Take. Both operands reuse topOperand, so a parameter is accepted where Access insists on a literal - EF passes the page bounds as @p, exactly what Jet's TOP cannot take. LimitNode gained the offset and its count became optional (OFFSET with no FETCH), and the sort bound is now offset + count, since the rows the sort would otherwise discard are the ones the skip consumes. Adding FIRST as a keyword broke the First() aggregate, caught by the suite. The grammar already had the fix pattern - functionName readmits LEFT/RIGHT/ASC for the same reason - so FIRST joins them. The other six new keywords were checked against the function surface and the .sql files: no collisions. CASE, both ANSI forms. Access has only IIF(), so the compat generator rewrites a CASE into nested IIFs. The simple form is folded into the searched one at parse time by rewriting each arm to `operand = value`, so evaluation sees one shape. Arms are tested in order, only true selects (a NULL condition is skipped like a false one), results short-circuit, and no match without ELSE is NULL. The T-SQL reference turned up two things beyond the syntax. Aggregates inside a CASE must be collected up front so they are computed per group - the standard specifies the same order - and neither HasAggregate nor Aggregates could walk into a CaseExpression, so HAVING CASE WHEN COUNT(*) > 1 failed with "Function COUNT is not supported". Conditions matter as much as results there. And CASE needed a declared type at all: the highest-precedence type across its branches, where an untyped branch (a bare NULL) contributes nothing rather than erasing the rest, numerics widen on the arithmetic ladder, and a genuine mismatch declares nothing rather than guessing. COALESCE - the first argument that is not NULL, NULL when all of them are. The standard makes it shorthand for a CASE over its arguments, so it takes the same type rule and reuses that helper. SQL Server implements the shorthand literally, which is why its own docs warn arguments are evaluated more than once and a subquery argument can differ between evaluations; each argument is evaluated at most once here, same answer without the instability. NULLIF's semantics were already right and already tested, but it had no declared type. It returns its first expression or a NULL of that expression's type, so unlike COALESCE it takes the first argument's type outright - the second only takes part in the comparison. The table value constructor, both of its uses. As an INSERT's VALUES clause it now takes a list of rows rather than one; the AST and executor were already row-list shaped, so only the grammar and builder needed it. DEFAULT is allowed as a row value, carried through as a sentinel rather than a value because it has to be distinguishable from NULL: on a column with a default, NULL stores NULL while DEFAULT takes the default. A column with no default stores NULL, and a required column with no default is refused rather than written as NULL. As a query - VALUES (1), (2) as an operand of a set operation - it is the shape EF emits for an inline collection, and its expressions may reference outer columns, so they are evaluated per outer row rather than folded once. Column names come from the leading query, per SQL, so no column alias list is needed; EF does not emit one, checked against the failure set. DEFAULT is confined to the INSERT clause, which the standard requires and the grammar gets for free by admitting rowValue nowhere else. A stored append procedure keeps its values as text, so it rejects both a multi-row constructor and a DEFAULT value - there is nothing to store for either. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Extended mode now emits the plain SQL form in the places the engine learned to handle in the previous commit, so the Jet-dialect rewrites come out of LibRedQuerySqlGenerator: the CASE-to-nested-IIF expansion, the COALESCE rewrite, and the surrounding null-guard scaffolding they needed. The compat generator in Common keeps every one of them - Access has no CASE, no COALESCE and no OFFSET, so that path still needs all of it. The postprocessor also stops running JetSkipTakePostprocessor in extended mode. That was the double-TOP-and-reverse emulation of a page, which existed only because Jet has TOP and nothing else; with native OFFSET/FETCH in the engine, EF's own paging form goes straight through. It joins the ORDER BY lift as the second pass gated to compat. Together these are what turn a query the generator used to rewrite into one it emits verbatim - which is the whole point of extended mode, and why the engine work had to land first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
71 files, from the run that took extended-mode failures from 16185 to 1696 against a compat baseline of 235. The arc, each step a single feature: 16185 after gating skip/take to compat, before the engine could page 5462 native ANSI paging - cleared ~10,700, the largest jump so far 2549 standard CASE - cleared ~2,900 1696 COALESCE and the table value constructor - cleared ~850 Paging dominates because 13844 of those 16185 failures carried "FETCH" in the error message: one missing grammar rule, not 13844 problems. Worth remembering as a triage habit - group the failures by error text before counting tests. The baselines only exist for tests that pass, since AssertSql runs after the query has executed and its results compared; anything still failing keeps its original placeholder. So these diffs are the newly-working queries, and the SQL in them is markedly plainer than the compat suite's: no nested TOP for a page, no IIF chain for a CASE, and far fewer derived-table layers now the ORDER BY lift is gone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Access has no CROSS JOIN keyword - a cartesian product is written there as comma-separated sources in the FROM clause - so the grammar had only the comma form, and joinClause made ON mandatory, leaving no shape for a join without one. EF Core's base generator emits the explicit keyword, so LibRed now takes both spellings for the same thing. Small change, because JoinKind.Cross was already plumbed through for the comma form: the planner, executor and index selection all handle a Cross join with a null condition already. CROSS JOIN builds the identical JoinTable node, so only the syntax was missing. joinClause splits into labelled alternatives to keep ON mandatory for the conditional kinds and absent for CROSS. That made JoinClauseContext abstract and broke the stored-view decomposition, which turned out to fit rather than fight: Access records a view's joins by their ON condition (Name1/Name2), so a CROSS JOIN contributes a source and no join entry - exactly how the comma form was already stored. The two spellings decompose identically, so nothing needed rejecting. CROSS was checked for identifier collisions before being made a keyword: no bare occurrence in any .sql file, and it is not a function or aggregate name. The tests pin the equivalence directly - CROSS JOIN and the comma form return byte-identical result sets - along with ON being rejected on a cross join and still required on a conditional one, so the new alternative did not loosen the existing rule. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The largest strip so far: a whole copy of the base VisitSelect, carrying five
unrelated Jet behaviours at once. Extended mode now uses the base method, so the
FROM clause is generated by GenerateFrom like every other provider.
What went with it, and what each was for:
parenthesised join nesting Jet is fussy about which join may follow which, so
VisitJetTables grouped them: FROM ((a JOIN b) JOIN c)
the DUAL pseudo-FROM a FROM-less SELECT needed a one-row table to select from
empty-projection substitution SELECT 1, or a column lifted out of the WHERE
the colexp IS NOT NULL guards appended to WHERE to restore inner semantics after
a join had been forced outer
comma joins Access's only cartesian product spelling
Four of those landed with no breakage at all. The entire cost was the fifth: base
GenerateFrom emits an explicit CROSS JOIN, which the grammar did not accept until
the previous commit. Failures went 1696 -> 2137 on the removal and 2137 -> 1693
once CROSS JOIN landed, so net -3 - but the count understates it, because the SQL
is markedly closer to standard. 1476 fewer parenthesised join groups, 355 LEFT
JOINs back to INNER, 308 fewer IS NOT NULL guards, 255 explicit CROSS JOINs.
The LEFT-to-INNER change is worth understanding: those joins were INNER to begin
with. EFCore.Jet manufactured a LEFT plus a null-rejecting WHERE guard to satisfy
Jet's join-ordering rules; nothing is being optimised now, the fabrication has
simply stopped. LibRed has no such rule - tableSource is a flat chain with a kind
per node, so a join following a join is just a join.
GenerateTop and GenerateLimitOffset stay. They are separate overrides and match
upstream SQL Server's exactly, which likewise has no VisitSelect override: TOP
for a bare limit, OFFSET/FETCH NEXT once there is a skip. TOP is not a Jet
workaround - it is native to the dialect LibRed implements, and the engine has
supported it from the start - so seeing it survive in the baselines is correct
rather than an oversight.
VisitJetTables remains for now, called only from VisitDelete and VisitUpdate,
which still emit Jet-shaped joins. The rebaseline confirms the split is clean:
zero DELETE or UPDATE lines moved across 69 files. Those two are a separate
change, after which the helper goes entirely.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
69 files. Every change is in the FROM clause - not one CASE, COALESCE, NULLIF,
#Dual, bracket placeholder or UNION line moved, and TOP/OFFSET/FETCH each shifted
by a single incidental line. Counted across the whole diff:
FROM ( 1676 -> 200 Jet's join parenthesisation, not derived tables:
`) AS` is 626 on both sides, so derived tables are
untouched. FROM ((a JOIN b) JOIN c) -> FROM a JOIN b JOIN c
INNER JOIN 618 -> 973 +355, exactly offsetting the LEFT JOIN fall
LEFT JOIN 2327 -> 1972 -355
IS NOT NULL 347 -> 39 -308, the guards that accompanied those joins
CROSS JOIN 0 -> 255 with comma-form FROM lines 127 -> 0
The three middle rows are one edit, not three. EFCore.Jet turned an INNER JOIN
into a LEFT JOIN plus a null-rejecting WHERE guard because Jet restricts which
join may follow which; base GenerateFrom simply emits the INNER JOIN that was
there all along.
DELETE and UPDATE are absent from the diff entirely - zero lines either side, and
RIGHT JOIN unchanged at 9 - because VisitDelete and VisitUpdate still route
through VisitJetTables. That path is a separate change and these baselines will
not move again until it lands.
Failures 1693, from 1696 before the strip.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Lateral joins: the right side is evaluated once per left row with that row
in scope, so it may correlate to the left. ACE has neither keyword, but EF
Core's base generator emits both, so extended mode needs them.
Grammar follows T-SQL's positioning, which lists
`left_table_source { CROSS | OUTER } APPLY right_table_source` as a
<joined_table> alternative next to the conditional joins and CROSS JOIN:
two more joinClause alternatives taking no ON, since the correlation inside
the right side is the condition.
Represented as two new JoinKind members rather than a separate node type, so
every kind-agnostic walker keeps working untouched and every optimizer site
that switches on kind - predicate pushdown, sort pushdown, index-nested-loop
and hash-join conversion - declines a lateral join by construction, because
each is an allowlist. Execution needs its own path regardless: ExecuteJoin
materialises the right side once against the enclosing scope, while
ExecuteApply re-runs it per left row with that row pushed onto the scope
chain, the mechanism a correlated subquery already uses. The joined schema
has to be known before any left row is read, so the right side is probed once
for its columns.
Two planner gaps surfaced with it, both of which made a nested APPLY
(Select_nested_collection_deep) look like a hang at ~62.7M rows:
- Place never descended into an APPLY. Both kinds preserve the left, so a
conjunct confined to it can be pushed there. Nothing is pushed into the
right side: under OUTER APPLY a filter there can empty an otherwise
non-empty result and manufacture the null-padded row the WHERE was
dropping.
- A conjunct naming an OUTER alias was never a pushdown candidate, because
Place demanded every qualifier be inside the subtree. PushPredicates now
takes the aliases the FROM introduces and ignores the rest, an outer column
being readable at any depth. General rather than APPLY-specific; it stayed
invisible while a correlated predicate still landed as a Filter directly
over its scan, which RewriteFilterOverScan already catches.
Index selection additionally plans a lateral right side with the left's
aliases as visible outer aliases, so a correlated predicate there becomes a
seek instead of a rescan per left row.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The AssertSql baselines these tests carried were SQL Server's, never ported, because JetCompatibilityExpressionVisitor rejected APPLY and the test utility turned that into a dynamic skip. With the visitor gated to compat mode the queries run, and the rewriter has now written what LibRed actually emits. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CVar(NULL) on the producing side and IIF(x IS NULL, NULL, CLNG(x)) on the consuming side were one workaround in two halves: Access loses the type of a bare NULL literal in a projection, so it was tagged on the way out and re-coerced on the way back in. LibRed carries declared types through the plan, so both halves go together - removing either alone would have been wrong. TryGenerateWithoutWrappingSelect no longer peeks a parent stack for an ambient InExpression; it tests directly for a select whose only table is a ValuesExpression, which is the condition it actually wanted. That was the stack's last live reader, so the stack, its five push/pop pairs, the GenerateIn override and _nullNumerics all go with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CVar(NULL) becomes NULL and the IIF/CLNG guard around the column reading it disappears. The bulk is TPC inheritance, which unions sibling tables and pads each absent column with NULL, so the wrapper appeared once per padded column per union arm. Also picks up the VALUES term now wrapped in its own subquery. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
EF Core's base generator emits ROW_NUMBER() OVER (PARTITION BY … ORDER BY …)
and the parser rejected it outright, which is the largest single block of
failures left in the extended suite. Access has no window functions at all, so
this is a LibRed extension.
Built so that adding another one costs a registry entry and nothing else. OVER
hangs off `functionCall` rather than off a list of window-function names, so
RANK, NTILE and friends already reach it through `functionName` at no grammar
cost, and `SUM(x) OVER (…)` parses for free. RANK and DENSE_RANK are included
to show that: four lines each, no grammar, no regeneration, no AST, no planner,
no executor.
PARTITION had to become a keyword for PARTITION BY, but Access has a real VBA
Partition(number, start, stop, interval) that LibRed implements and tests, so
it is readmitted as a function name beside FIRST - a call is always followed by
'(' and PARTITION BY never is. OVER is a plain new reserved word.
WindowFunction is deliberately NOT a FunctionCall subtype: HasAggregate matches
any FunctionCall whose name is an aggregate, so a windowed aggregate would make
the query look grouped and build a bogus AggregateNode.
The planner lifts each call out of the projection into a reference to a
synthetic column WindowNode publishes, so DeclaredType, ProjectionSchemaFor,
sorting, DISTINCT and LIMIT all keep seeing an ordinary column and needed no
changes. WindowNode preserves input order - the sort sits above it, so emitting
in partition order would silently reorder any windowed query with an ORDER BY.
Two optimizer sites needed care rather than luck. IndexSelection.Apply's
default arm stops descending, not just rewriting, so without a WindowNode case
every windowed query would silently become a full scan - correct results,
catastrophic plans, the failure mode that made nested APPLY look like a hang; a
plan-shape test guards it. SubtreeAliases needed the same pass-through case
ProjectNode's comment records having needed. PushPredicates needs nothing: the
WindowNode is built after pushdown has run.
A window over a grouped query throws rather than silently misbehaving:
AggregateNode owns the projection and collapses rows, and nothing EF emits
needs it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Window functions now execute, so the ~180 tests that emitted ROW_NUMBER() OVER(PARTITION BY … ORDER BY …) run for the first time and their baselines are written from LibRed output rather than the SQL Server text the project inherited when it was copied. Multiple_members_of_correlated_single_result_subquery_lift_to_single_join had its `method switch` flattened by the rewriter, which can only write one baseline per test - so the First/Single arm was kept and the Last and ElementAt arms were lost, leaving 8 of its 16 cases failing. The switch is restored with each arm's real output: Last orders the window DESC, ElementAt uses the two-sided 0 < row AND row <= 1 predicate. It is the only test in either suite whose AssertSql sits inside a switch, so the exposure is that one test. Checked before committing: nothing a LibRed run cannot produce appears on the added side - no SQL Server TOP(n), no N'' literals, no @__ parameter names, and the single bracketed line is raw SQL the keyless-entity fixture supplies as input. All 9 removed APPLY baselines are stale (6 SQL Server, 3 Jet-era). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The hook payload carries "scratchpad_dir" on every call, and the exemption (scratchpad|\.log|\$log|/tmp/) was grepped against the whole payload - so it matched unconditionally and the read guard exempted itself on every command. It has never fired since it was written. Checks now run against the payload with the envelope fields stripped (scratchpad_dir, transcript_path, cwd, and description, so model-written prose cannot trigger or exempt a rule either). The two hooks are merged into one that reads stdin once. They both consumed stdin independently, which happened to work but meant a third check could break the second; one read removes the ordering dependency. A deny-only hook leaves everything else to the permission system, so a legitimate scratchpad read still cost a prompt. The exempt case now returns an explicit allow. Because that grants rather than blocks and the exemption is a text match, the allow branch declines when the command contains ; & ` or $( , so a chained command merely mentioning scratchpad falls through to a prompt instead of being auto-approved. Plain pipes still qualify. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`x IN (SELECT … UNION ALL SELECT …)` failed to parse: IN, EXISTS and a scalar subquery each took a bare selectStatement, while a derived table has always taken a full queryExpression. The standard reaches all four positions through the same <query expression> nonterminal, so this was an internal inconsistency rather than a deliberate narrowing. EF Core emits the flattened form once its generator elides the wrapping select it would otherwise put around the union - which 51a2b24 enabled by removing the parent-stack peek that had been forcing that wrapper inside an IN. The generator is right and the engine was behind. The three AST records now carry SqlStatement, matching SubqueryTable, rather than the parse being desugared into a synthetic derived table. That makes the compiler enumerate every consumer instead of leaving them to be found by inspection - it turned up AggregatesInSelect and ViewExpander, neither of which was on the list drawn up beforehand. Consumers split two ways. Those that only run the body take it as it comes: SubqueryPlan now calls PlanStatement rather than PlanSelect. Those that inspect its shape - the EXISTS, IN and scalar-aggregate decorrelation rewrites - read a projection, FROM and WHERE that a set operation does not have, so each call site now declines anything that is not a plain SELECT. Declining costs speed, never correctness: the per-row path runs the body as written. AggregatesInSelect is the exception and walks both arms, because an aggregate over an outer column left uncollected would fail to resolve rather than merely run slower. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`SELECT … UNION SELECT … ORDER BY x` returned the rows in the wrong order and said nothing: ORDER BY and OFFSET/FETCH lived on selectStatement, so a trailing one bound to the LAST OPERAND. Measured against ACE with the probe added here - over UA(50,10) and UB(40,20), ACE returns 10,20,40,50 where LibRed returned 50,10,20,40, which is UA unsorted followed by UB sorted. The grammar now states where the clauses may appear rather than a builder correcting it afterwards. selectStatement becomes an order-less querySpecification - the standard'"'"'s own name for that rule, and Trino'"'"'s - and queryExpression carries `orderByClause? offsetFetchClause?`, which is the standard'"'"'s structure: <query expression> is ordered, <query term> is not. Every serious SQL grammar factors it this way (Trino, SQLite'"'"'s select_core, PostgreSQL'"'"'s simple_select) for the reason met here, that an operand rule ending in an optional ORDER BY swallows the enclosing expression'"'"'s greedily. That also removes an ambiguity rather than working around it: SelectStatement.Top holds either a leading TOP or a trailing FETCH, and lifting the right one used to need a peek at the parse tree. Now a leading TOP is on the operand'"'"'s specification and a trailing FETCH on the expression, and they cannot be confused. SetOperationStatement carries the clauses; the planner puts SortNode/LimitNode above the set operation. A single term folds them back onto its SELECT, so an ordinary `SELECT … ORDER BY x` builds exactly the AST it always did - pinned by a test, since that is the case that would ripple everywhere if it changed. Two consequences worth knowing. An operand may still be ordered by parenthesising it, and must be: `(SELECT TOP 5 … ORDER BY x) UNION …` is a nested query expression, and the ordering is what makes that TOP deterministic. And ORDER BY on a non-final operand is now a parse error where ACE parses and ignores it - a deliberate divergence, since ACE ignoring it means no such query ever depended on it, and refusing loudly beats accepting silently.
EF emits a correlated COLUMN as the skip for `ElementAt(<column>)`: `OFFSET `s`.`Id` ROWS FETCH NEXT 1 ROWS ONLY`. offsetFetchClause reused topOperand, which admits only a literal, a parameter or a parenthesised expression, so that failed to parse. The two rules are no longer shared, because the restriction was never about paging. TOP has to stay narrow: it sits immediately before the select list, where an unrestricted expression would swallow the star of `SELECT TOP 5 * FROM t` - the grammar already said so. A paging count is closed by the ROW/ROWS keyword that must follow it, so there is nothing there to swallow and it can be a full expression. Wider than what is written down, and deliberately so. The standard'"'"'s <offset row count> is a <simple value specification> - literal, parameter or variable - and SQL Server documents offset_row_count_expression as a variable, parameter or constant scalar subquery; a correlated column is none of those. But SQL Server accepts one, which EF Core'"'"'s own SQL Server baseline for Where_subquery_with_ElementAt_using_column_as_index demonstrates: it carries `OFFSET [s].[Id] ROWS`, and exists only because that test passed against a real server. This matches the engine, not the docs. Nothing was needed in the executor: LimitNode already evaluates its bounds against the enclosing scope, which is where a correlated column resolves - and only there, since the bounds are evaluated with an empty row schema, so a reference to the paged query'"'"'s own columns cannot resolve at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four files move. The GearsOfWar trio picks up `x IN (<union>)` now that a subquery predicate takes a full query expression, so the wrapping select EF used to be forced to emit is gone from the baselines. PrimitiveCollections picks up the correlated-column OFFSET, and is worth a look as the case where several of these compose: a set operation mixing a SELECT with a VALUES term, used as a correlated derived table, with an outer column inside a row value, ordered, and paged by `OFFSET `p`.`Int` ROWS`. It also shows the column-alias-list decision holding - SQL Server writes `AS [v]([_ord], [Value])` where EF names the columns from the leading SELECT of the union for us, which is exactly why that list was never needed. Checked before committing: nothing a LibRed run cannot produce appears on the added side - no SQL Server brackets, TOP(n), N'' literals or @__ parameters - and the OFFSET and UNION lines are replaced one for one rather than lost. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New
Both: Add new Parse translator
Both: Implemented a jet compatible NULLIF to IIF... sql translation
LibRed only
Full BMP plane coverage of General and General Legacy collating orders
Implemented tailorings for all 30 non CJK collating orders
Database can be created with a specific collating order
Implemented INSERT INTO ... SELECT
Implemented SELECT ... INTO
Implemented NULLIF function
Implemented FULL JOIN
Bug Fixes
Continue the Autonumber counter past the int32 wrap
Fix the migrations-lock hang in JetHistoryRepository
Take the relocation pointer from the slot's first 4 bytes, not its width